mean-field approximation
Cluster Variational Approximations for Structure Learning of Continuous-Time Bayesian Networks from Incomplete Data
Neural Information Processing Systems http://nips.cc/
On the detrimental effect of invariances in the likelihood for variational inference
Variational Bayesian posterior inference often requires simplifying approximations such as mean-field parametrisation to ensure tractability. However, prior work has associated the variational mean-field approximation for Bayesian neural networks with underfitting in the case of small datasets or large model sizes. In this work, we show that invariances in the likelihood function of over-parametrised models contribute to this phenomenon because these invariances complicate the structure of the posterior by introducing discrete and/or continuous modes which cannot be well approximated by Gaussian mean-field distributions. In particular, we show that the mean-field approximation has an additional gap in the evidence lower bound compared to a purpose-built posterior that takes into account the known invariances. Importantly, this invariance gap is not constant; it vanishes as the approximation reverts to the prior. We proceed by first considering translation invariances in a linear model with a single data point in detail. We show that, while the true posterior can be constructed from a mean-field parametrisation, this is achieved only if the objective function takes into account the invariance gap. Then, we transfer our analysis of the linear model to neural networks. Our analysis provides a framework for future work to explore solutions to the invariance problem.
Cluster Variational Approximations for Structure Learning of Continuous-Time Bayesian Networks from Incomplete Data
Neural Information Processing Systems http://nips.cc/
aa1f5f73327ba40d47ebce155e785aaf-AuthorFeedback.pdf
We would like to thank all the reviewers for their thoughtful comments and their enthusiasm for our work. These results are consistent with those of Zoltowski et al. [2020], where they found Laplace EM compared Section 3. Segmenting the continuous latent states for each population (which is equivalent to imposing hard constraints On top of that, the "sticky" parameterization of discrete state transitions reveals which neural populations C. elegans offers an illustrative demonstration of the mp-srSLDS For example, we explore interactions between ganglia in Appendix C. Thanks again for spending the time to provide valuable feedback on our work.
Revisiting Logistic-softmax Likelihood in Bayesian Meta-learning for Few-shot Classification
Furthermore, we theoretically and empirically show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax.